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Abstract 

Programs serving infants and toddlers are expected to use child data to inform decisions 
about intervention services; however, few tools exist to support these efforts. The Making 
Online Decisions (MOD) system is an adaptive intervention that guides early educators’ 
data-based intervention decision making for infants and toddlers at risk for language delay. 
Using a cluster randomized design to test the effect of the MOD, home visitors (HVs) were 
assigned to either use the MOD or not across |3 Early Head Start programs. Both groups 
used the Early Communication Indicator (ECI) for progress monitoring and a parent-mediated 
language promotion intervention. Children from both groups demonstrated significant growth 
in expressive communication. However, children whose HVs fully implemented the MOD grew 
significantly more than the group that did not use the MOD, even after statistically controlling for 
parent and HV variables. Implications for designing effective and usable systems to promote the 
use of data-based decision-making practices by infant—-toddler service providers are discussed, 
as well as limitations of the current study. 
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Introduction 


Early language and literacy experiences prior to kindergarten are key to children’s later language 
development (Hart & Risley, 1995), early literacy skills, and overall readiness for school (Lonigan 
& Shanahan, 2009). Children’s earliest language learning opportunities predict reading achieve- 
ment and school readiness in elementary grades (e.g., Hoff, 2013; Walker, Greenwood, Hart, & 
Carta, 1994), as well as academic outcomes in later school years (Snow, Burns, & Griffin, 1998; 
Whitehurst & Lonigan, 1998). Furthermore, early language skills are positively correlated with 
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measures of social competency (Doctoroff, Greer, & Arnold, 2006; Miles & Stipek, 2006). 
Unfortunately, children whose early experiences provide them few opportunities to hear and 
practice language struggle to develop preliteracy and communication skills and are therefore less 
likely to benefit from reading instruction (e.g., Dickinson, Griffith, Golinkoff, & Hirsh-Pasek, 
2012). Ensuring that children experience high-quality language experiences during their first 
years of life is an important preventive factor in reducing the need for special education services, 
preventing academic problems, and reducing school dropout (Burchinal et al., 2000; Payne, 
Whitehurst, & Angell, 1994). 

The National Early Intervention Longitudinal Study (Hebbeler et al., 2007) reported that 
41.1% of children who entered early intervention services had a communication delay, and for 
most, those problems continued through kindergarten (Hebbeler & Gerlach-Downie, 2002). The 
prevalence of communication problems is widespread and can result in significant personal and 
national cost (e.g., Kaiser, Hancock, & Nietfeld, 2000; Walker & Carta, 2010). 


Delivery of Early Intervention Through Home Visiting 


Home visiting is an early intervention delivery model that has experienced substantial expansion 
over the past decade. Indeed, most programs that serve infants and toddlers with and without 
special needs (e.g., Part C of Individuals with Disabilities Education Act [IDEA], Early Head 
Start [EHS], and Parents as Teachers) include some form of home visiting services on a recurring 
basis. Thus, the provision of evidence-based practices (EBPs) in the home by home visitors 
(HVs) is of great importance. Unfortunately, HVs and others providing early intervention often 
have limited access to resources needed to effectively individualize home-based services to chil- 
dren and promote parent engagement in intervention delivery (Dunst, Trivette, & Raab, 2013). 
HVs need training on the use of EBPs because they are expected to know how to individualize 
curriculum and use assessment data to meet children’s needs and maintain parent engagement 
(Administration for Children and Families, Office of Head Start 2015). 

The effectiveness of home visiting varies depending on the model used, population served, 
and targeted outcomes (Astuto & Allen, 2009; Boyd, Odom, Humphreys, & Sam, 2010; Raikes 
et al., 2006). In a systematic review of randomized control trials (RCTs) testing the efficacy of 
paraprofessional home visiting programs, Peacock, Konrad, Watson, Nickel, and Muhajarine 
(2013) reported that improvements in child cognition and behavior, as well as reductions in child 
maltreatment, were strongest when home visits began prenatally and occurred frequently. In 
addition, Dunst, Bruder, and Espe-Sherwindt (2014) reported that parents were more likely to be 
involved in their child’s early intervention when services were provided in the home. Kahn and 
Moore (2010) reported that of 66 home visiting programs evaluated in RCTs, 32 had significant 
effects in at least one child domain, 23 had mixed outcomes (e.g., short-term effect that did not 
sustain to follow-up or an effect on one subpopulation but not another), and 11 did not have a 
significant effect on any child outcomes. 

Because home visiting occurs in a wide range of settings relative to other models in more 
consistent settings (e.g., child care centers, classrooms, hospitals, and clinics), there are multiple 
factors that can affect the fidelity, dosage, and, ultimately, the effectiveness of home visiting 
services. For example, there is evidence that strong parent—HV relationships in which parents 
respect HVs’ advice and HVs balance programmatic fidelity with parents’ needs and emotional 
well-being are linked to critical variables such as the quality of services families receive and cli- 
ent retention (Barak, Spielberger, & Gitlow, 2014; Berlin, O’Neal, & Brooks-Gunn, 1998; 
Korfmacher, Green, Spellmann, & Thornburg, 2007; McCurdy & Daro, 2001). In addition, dif- 
ferences in training requirements between early childhood programs and states have resulted in 
variation in the education backgrounds of HVs serving families of young children (Gill, 
Greenberg, Moon, & Margraf, 2007; Harden, Denmark, & Saul, 2010). Burchinal and her 
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colleagues (2002) found that early childhood professionals with more preservice and in-service 
training had higher quality classrooms, had children with higher language and preliteracy out- 
comes, and were rated higher by parents than educators with less education and training. Finally, 
as the primary language of the U.S. population continues to diversify, there is growing concern 
about the effectiveness of EBPs in which the influence of language background has not been 
examined (Buzhardt, Rusinko, Heitzman-Powell, Trevino-Maack, & McGrath, 2016; Valdez, 
Ramirez-Stege, Martinez, D’Costa, & Chavez, 2017). 


Data-Based Decision Making to Guide Early Intervention 


Current program standards call for the use of individual child progress data to inform early 
intervention decisions and practices, including EHS (45 CFR Sec 1302.33(b) (2)) and Part C 
of the IDEA. Also, Response to Intervention (RTI) and Multi-Tier System of Support (MTSS) 
models used in early childhood programs call for data-based decision making (Carta et al., 
2014, 2016; Division for Early Childhood of the Council for Exceptional Children, National 
Association for the Education of Young Children, & National Head Start Association, 2014). 
Examples of data sharing in programs serving infants and toddlers in EHS have been reported 
(e.g., Guss et al., 2013; Yazejian et al., 2017). Models that use data-based decision making to 
individualize early intervention services include the Pyramid Model (Hemmeter, Fox, & 
Snyder, 2013), Building Blocks (Sandall & Schwartz, 2013), and Recognition and Response 
(Buysee et al., 2013). 

Although universal screening and progress-monitoring measurement systems are available for 
use in data-based decision-making models, including Infant and Toddler Individual Growth and 
Development Indicators (IGDIs), myIGDIs for preschool, and DIBELS for K-3 (Greenwood, 
Carta, & McConnell, 2011), many early childhood programs struggle to find and use data-based 
decision making in a way that makes a meaningful difference in child outcomes (Akers et al., 
2015). These limitations extend to home visiting service models wherein parents and other fam- 
ily members are expected to implement HV recommendations in the home (Azzi-Lessing, 2011; 
Bruder, Dunst, Wilson, & Stayton, 2013; Dunst et al., 2014). In addition to a lack of in-service 
and preservice training, one reason for the limited use of data-based decision-making practices in 
infant—toddler programs may be the perceived effort and cost of collecting and analyzing out- 
come data at the expense of other, more direct family services. Indeed, there is evidence that as 
interventions become more complex (e.g., adding data-based decision making), they are less 
likely to be implemented with high fidelity because of a higher number of potential barriers 
(Carroll et al., 2007; Dusenbury, Brannigan, Falco, & Hansen, 2003) and that an intervention’s 
complexity negatively affects fidelity, sustainability, and acceptability by practitioners (Bywater, 
2014; Carroll et al., 2007). 


The Making Online Decisions (MOD) System for Infant—Toddler Programs 


One tool that supports HVs’ use of child data to individualize parent-mediated intervention is 
the MOD system (Buzhardt et al., 2010; Buzhardt, Walker, Greenwood, & Carta, 2011) for use 
with infant and toddler IGDIs (Carta, Greenwood, Walker, & Buzhardt, 2010). The MOD sys- 
tem, described further in the “Method” section, is an online “adaptive intervention support 
system.” In their review of adaptive interventions used in child mental health services, Almirall 
and Chronis-Tuscano (2016) describe this form of intervention as providing “a replicable guide 
for the provision of individualized sequences of interventions in actual clinical practice” (p. 
383). To our knowledge, the MOD system is one of the few adaptive interventions designed 
specifically to guide infant—toddler service providers’ use of child data to individualize inter- 
vention services. 
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The MOD was effective in a prior study (Buzhardt et al., 2011) in which HVs were randomly 
assigned to either use the MOD to guide their use of progress-monitoring data and the Strategies 
for Promoting Communication Tools for Accelerating Language in Kids: PC TALK intervention 
(Walker & Bigelow, 2012) or not. All HVs used the Early Communication Indicator (ECI), a 
brief, play-based infant—toddler assessment (see the “Method” section for more details about this 
measure), to monitor children’s growth in expressive communication. When a child scored less 
than 1 SD below benchmark on the ECI, HVs using the MOD system were guided through a data- 
based decision-making model, which included recommended Promoting Communication Tools 
for Accelerating Language in Kids (PC TALK) strategies based on children’s ECI performance. 
HVs in the comparison group also used the ECI to monitor progress and had access to the PC 
TALK strategies, but did not use the MOD to guide them or automatically select appropriate 
strategies to share with parents. Results indicated that low-performing children whose HVs used 
the MOD demonstrated significantly stronger growth in expressive communication than children 
whose HVs did not use the MOD (p < .05). Furthermore, the effect size increased from d = 0.47 
after 6 months of intervention to d = 0.71 after 9 months of intervention. 

Although these results were promising, given the limited nature of this study (e.g., small 
sample size in a single state) and the lack of other empirical research to draw upon, additional 
questions remain about HVs’ use of this tool and their data-based decision-making practices in 
general. A primary limitation was a lack of data on HVs’ implementation fidelity (dose-response 
relationships) and the degree to which full implementation of the MOD framework affected child 
outcomes compared with partial implementation. This is especially relevant to the MOD in this 
context because HVs used it only with a small portion of their caseload, if at all (i.e., only those 
children who scored at least 1 SD below benchmark on the ECI). Therefore, because it was not a 
regular part of their protocols with all families, there is a risk that implementation becomes 
irregular when they do use it. Also, the influence of other factors discussed earlier is unknown, 
including the working relationship between HVs and parents, HV and parent education back- 
ground, and families’ primary language. Finally, because an intervention’s sustainability is influ- 
enced by its usability and relevance to practitioners’ work (Scheirer, 2005), measures of HVs’ 
satisfaction with the MOD are needed. 


Purpose and Research Questions 


We conducted a conceptual replication of our earlier study to address prior limitations, focusing 
on the following research questions: 


Research Question 1: What is the effect of the MOD on infants’ and young children’s growth 
in expressive communication? 

Research Question 2: What is the effect of HVs’ level of MOD implementation on parent- 
reported HV support and children’s growth in expressive communication? 

Research Question 3: Do HV education, parent education, home language, and parent-HV 
relationships moderate the effect of the MOD on child outcomes? 

Research Question 4: What is the level of HVs’ reported satisfaction with the MOD 
intervention? 


Method 


Design 


We used a cluster randomized design to evaluate the effects of MOD on low-performing chil- 
dren’s growth in expressive communication. Clusters were the children nested under their HV. To 
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to individualize PC Talk PC TALK based on ECI scores 


Treatment Phase 
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Ongoing ECI | Ongoing ECI | 
Assessments Assessments 


Figure I. Sequence of study activities, including HV training, randomization, decision-making approach, 
and ongoing monitoring of children’s expressive communication. 
Note. HV = home visitor. 


control for differences between EHS programs, HVs were randomly assigned within programs to 
experimental conditions: Evidence-based Practice Only (EBP Only) or Evidence-based Practice 
+ MOD (EBP + MOD). Figure 1 illustrates the sequence of recruiting, training, and randomiza- 
tion of HVs to conditions. 


Participants 


EHS programs. Thirteen EHS programs from four Midwestern states providing weekly home 
visits to families participated. EHS is a federally funded, comprehensive, two-generation 
program aimed at enhancing the development of infants and toddlers while strengthening 
families (U.S. Department of Health and Human Services, Administration for Children and 
Families, 2015). EHS programs serve families with incomes at, or below, the federal poverty 
level, and at least 10% of the available child openings in EHS are reserved for children with 
special needs. 


EHS HVs. Although requirements for EHS staff vary across programs, legislation requires staff 
who provide services to children and families to have a “minimum of a Child Development Asso- 
ciate (CDA) or comparable credential, and have been trained or have equivalent coursework in 
early childhood development” (Sec. 645A(h)). Table 1 describes the characteristics and group 
assignment of the 45 HVs who participated. 


Children/families. Only children who scored at least 1 SD below their age-based benchmark on the 
ECI were eligible to participate in the present study. Of the 296 families with children below 
benchmark, 146 families (49.3%) consented and participated. Eighty children participated in the 
EBP-Only group (55% male, 45% female). Their age at benchmark eligibility was 21.3 months 
(SD = 7.6). Sixty-six children (65% male, 35% female) participated in the EBP + MOD group; 
mean age at benchmark eligibility was 20.5 months (SD = 6.7; see Table 1). 
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Table |. Participant Demographics. 


EBP Only EBP + MOD 

HVs N= 22 N = 23 
Gender 

F 21 (49%) 23(51%) 

M 1 (100%) 0 (0%) 
Ethnicity 

White/Caucasian 20 (51%) 19 (49%) 

Hispanic 2 (60%) 2 (40%) 

Other 0 (0%) 2 (100%) 
Education 

HS degree or GED | (50%) | (50%) 

Some college, up to BA degree 19 (51%) 18 (49%) 

Some graduate school, up to MA 2 (40%) 3 (60%) 

Mean caseload size (SD) 10 (3.35) 11 (1.48) 

Mean number of children with language delay (SD) 3 (2.62) 4 (2.70) 

EBP Only EBP + MOD 

Families N = 80 N = 66 
Primary caregiver highest education level 

Some high school 13 (39%) 20 (61%) 

HS degree or GED 33 (60%) 22 (40%) 

Some college, up to BA degree 33 (61%) 21 (39%) 
Primary home language 

English 71 (58%) 51 (42%) 

Spanish 7 (33%) 14 (67%) 

Other 2 (100%) 0 (0%) 
Household monthly income 

Up to US$1000 47 (56%) 37 (44%) 

US$1,001-US$2,000 23 (51%) 22 (49%) 

US$2,001-US$3,000 4 (57%) 3 (43%) 

Over US$3,000 1 (100%) 0 (0%) 
Marital status 

Single 30 (65%) 16 (35%) 

Married 50 (51%) 49 (49%) 
Mean additional children in home (SD) 2 (1.58) 2 (1.56) 
Mean number of adults in home (SD) 2 (1.05) 2 (0.98) 


Note. HV= home visitor; EBP = Early Head Start; MOD = Making Online Decisions; HS = high school; 


GED = General Educational Development. 


HV training. All HVs were trained and certified to administer and score the ECI (see the Measure- 
ment section for description of the ECI) for all children on their caseload. As part of standard 
protocol for Infant and Toddler IGDIs (Walker & Buzhardt, 2010), ECIs were administered quar- 
terly for all children and more frequently for children scoring at least 1 SD below benchmark 
(e.g., monthly or bimonthly). HVs entered children’s ECI scores into the secure, password-pro- 
tected IGDI Online Data System (Buzhardt et al., 2010) (www.igdi.ku.edu) for processing. 
Reports and graphs were then available to view children’s growth in expressive communication 


(Greenwood, Walker, & Buzhardt, 2010). 
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Using a standard protocol (Walker & Buzhardt, 2010), all HVs received ECI training in a 
1-day workshop format. The workshop included discussion, practice, and feedback on ECI scor- 
ing and administration, including how to use the Online Data System. To become certified on the 
ECI, HVs had to score two full ECI assessments from video and achieve at least 85% agreement 
with master scorings of those videos by ECI developers. Using the ECI Administration Checklist 
(Walker & Buzhardt, 2010), one HV from each program submitted a video of an ECI administra- 
tion to trainers, who checked that she completed at least 13 of 16 administration tasks and pro- 
vided the HV with feedback. That HV then checked fidelity of the other HVs in her program. Key 
items on the administration checklist are those regarding the play partner’s behavior during the 
play session: following the child’s lead, commenting about what the child is doing, describing 
what the child is doing, being nondirective and friendly with the child, and using questions spar- 
ingly. HVs received refresher trainings annually that included scoring an ECI at 85% agreement 
on different ECI videos from those used in the original training. 

Also, all HVs were trained in the use of the PC TALK intervention (Walker & Bigelow, 2012) 
for families (www.talk.ku.edu). Initial PC TALK training lasted about an hour and was also part 
of the annual refresher trainings. PC TALK is a manualized set of evidence-based intervention 
strategies and supporting tools derived from milieu and responsive teaching techniques (e.g., 
Kong & Carta, 2011; Roberts & Kaiser, 2011; Warren & Brady, 2007). The PC TALK strategies 
were designed to extend parents’ and caregivers’ use of language-promoting strategies across 
daily routines (Walker, Bigelow, & Harjusola-Webb, 2008). PC TALK strategies included (a) 
arranging the environment, (b) following the child’s lead, (c) commenting and labeling, (d) imi- 
tating and expanding, (e) open-ended questioning, (f) time delay, (g) positive attention, and (h) 
providing choices. Each strategy is indexed to infants, toddlers, and young children at differing 
stages of communication development. The PC TALK manual provided background description 
as to the purpose of the strategies, importance, and how to embed them in routines (e.g., meals, 
book reading). HVs randomly assigned to the EBP + MOD group also received a |-hr group 
training on how to use the MOD (e.g., complete MOD steps, enter checklists, and print MOD- 
recommended PC TALK strategies). 


Experimental Conditions 


EBP-Only HVs. EBP-Only HVs individualized the PC TALK strategies for parents based on their 
child’s ECI scores. Trainers showed HVs how to use children’s ECI key skill element sores 
obtained from the online data system to decide which strategies to recommend. For example, 
they recommended basic strategies for children whose data indicated they were using gestures 
and vocalizations, but not words or phrases, and more advanced strategies for children whose 
data showed them beginning to use words. Other than annual trainings, no further guidance or 
follow-up was provided to HVs related to the PC TALK intervention or ECI alignment to the 
intervention. 


EBP + MOD HVs. In addition to using the ECI and the PC TALK strategies, EBP + MOD HVs 
used the MOD to guide their use of the ECI assessment and the PC TALK strategies with parents. 
Following initial training, a research assistant activated the MOD within the IGDI online data 
system, and the MOD automatically began providing guidance for an HV, as described below, 
when she or he entered an ECI for a child that was at least 1 SD below benchmark. Based in part 
on Tilly’s (2002) problem-solving approach, EBP + MOD HVs completed the following steps 
for children performing below benchmark on the ECI: 


1. Js there a problem? Children who score at least 1 SD below their age-based norm on the 
ECI are identified as not making adequate progress, and the data system immediately 
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displays the child’s progress-monitoring graph. The MOD asks the HV to indicate 
whether the ECI was a valid assessment. HV action: If the ECI was not valid, then the HV 
documents within the MOD reason(s) why the administration was not valid, and the 
application recommends repeating the assessment within 2 weeks. If it was valid, then the 
MOD moves to Step 2. 

2. What is causing the problem? If the HV confirms that the low ECI is valid, then the MOD 
asks the HV three questions about why the child’s language is below benchmark: (a) Does 
the HV believe there may be medical concerns (e.g., hearing impairment)? (b) Have there 
been recent changes in the child’s home (e.g., recent separation or change in primary resi- 
dence)? (c) Have there been recent changes in the child care environment (e.g., new child 
care center, staff turnover)? HV action: The HV documents in the MOD any concerns she 
believes may be causing the child to perform below benchmark. Then, the MOD recom- 
mends more frequent ECI assessments to monitor the child’s progress more closely and 
continues immediately to Step 3. 

3. What intervention should be used? The MOD summarizes the child’s ECI performance 
and uses algorithms to recommend specific PC TALK strategies for parents to use based 
on the child’s current performance on ECI key skills (gestures, vocalizations, single 
words, and multiple words). HV action: From the strategies recommended by the MOD, 
the HV selects strategies based on her experience and collaboration with the family to 
share with parents/caregivers. From the MOD, the HV prints the selected PC TALK strat- 
egies, which includes their definition, why they are important, and ways that parents can 
use them during daily routines, and takes this information to review with parents. 

4. Is the intervention being done? The MOD provides Fidelity Checklists to guide and docu- 
ment how HVs introduce the PC TALK strategies to parents and provide families with 
ongoing support to integrate the strategies into families’ daily routines. HV action: Over 
subsequent weeks, HVs introduce the strategies to parents/caregivers, describe how to 
use them, model their use with the child, and provide feedback to parents about their use 
of the strategies based on discussions with the parents and/or the HVs’ observations of the 
parents. HVs enter their completed MOD Fidelity Checklists into the MOD to document 
their work with parents and to inform next steps. See Buzhardt and colleagues (2011) for 
a complete list of checklist items. 

5. Is the intervention working? After three more ECIs are completed for the child, the MOD 
analyzes and reports to the HV the child’s communication growth since beginning the PC 
TALK strategies. The analysis considers three dimensions of the child’s ECI perfor- 
mance: the most recent ECI total score compared with the score before Step 3, the slope 
(average rate of growth) of the ECI before and after PC TALK, and the child’s predicted 
progress 6 months later based on current score and slope. HV action: The HV uses these 
analyses to inform recommendations to parents about their ongoing use of the PC TALK 
strategies. For example, given no improvement, the MOD suggests considering more 
intensive interventions or additional services (e.g., speech language pathologist), and the 
cycle continues. Also, the MOD updates the recommended strategies based on new data 
and encourages the HV to continue working with the parents on use of the strategies. 


After Step 5 begins, the MOD continues to provide HVs with updated analysis of the child’s 
growth after each additional ECI is entered regardless of the child’s age, length of time in the 
MOD, or ECI performance. Therefore, as an adaptive intervention, the amount of time an HV 
uses the MOD for a child depends on the age at which the child scores below benchmark (recom- 
mendations end at 42 months) and how long the family continues to receive intervention services 
from a service provider using the MOD. For a more comprehensive description of the MOD, see 
Buzhardt and colleagues (2010). 
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Measurement 


ECI. The ECI (Buzhardt & Walker, 2010; Greenwood et al., 2010; Walker et al., 2008) is a 6-min 
play-based observational measure of a child’s communication during a standard play session with 
a familiar adult. Designed for use by early childhood practitioners for universal screening and prog- 
ress monitoring, its monthly age-based benchmarks inform intervention decision making for indi- 
vidual children 6 to 42 months of age (Greenwood et al., 2010; Greenwood, Buzhardt et al., 2011) 
and program-level improvement (Greenwood, Buzhardt, Walker, Howard, & Anderson, 2011). 

The ECI was adopted by participating EHS programs and used with all children in the study. 
Two standard toy sets are used, the Fisher Price® House and Barn, as the play context for each 
administration. The House and Barn serve as alternate forms and are used on alternate measure- 
ment occasions. A parent or the child’s primary caregiver served as the play partner, whereas the 
HV either scored the ECI in person or video-recorded it to score later. The HV reviewed the 
guidelines for play partners (e.g., following child’s lead, commenting about the child’s play) 
prior to assessments and “practiced” with the child prior to the parent’s first assessment, provid- 
ing feedback as needed. After HVs scored an ECI, they entered the scores into the secure online 
data system that provided automated scoring and graphing of individual child and program-level 
data (www.igdi.ku.edu). The online data system calculates a rate per minute for each of four key 
skills (gestures, vocalization, single words, and multiple words) and a weighted total score that is 
the sum of the key skills with words multiplied by 2 and multiple words multiplied by 3. 

Reports of the ECI’s psychometrics indicate an interobserver agreement index of .90, test- 
retest reliability of .89 for mean level, .62 for slope, alternate forms reliability of .72, and crite- 
rion validity of r = .62 referenced to the Preschool Language Scale (Zimmerman, Steiner, & 
Pond, 2011). The construct validity of the ECI’s continuum of key skills trajectories over time 
was established in a large sample (Greenwood, Buzhardt, Walker, McCune, & Howard, 2013). 
The equivalence (invariance) of the ECI’s measurement properties (key skills and composite 
scores) was demonstrated comparing two time-displaced samples of ECIs, all collected by prac- 
titioners (Greenwood et al., 2013). The ECI also has monthly age-based growth benchmarks 
based on the composite score mean up to 42 months of age (Greenwood et al., 2010). 


MOD implementation stages. The completion of each MOD step resulted in a permanent record in 
the IGDI online data system that was later retrieved for analysis. HVs’ MOD implementation was 
grouped into three implementation levels: Stage 1—MOD Steps 1 to 3, Stage 2—Steps | to 4, 
and Stage 3—Steps | to 5. The rationale for these levels is that progression to each subsequent 
level represents a meaningful advancement in the data-based decision-making model. For exam- 
ple, we grouped Steps 1 to 3 into Stage 1 because Step 3 is the point at which the HV selects 
MOD-recommended PC TALK strategies. 

At Stage 2 implementation, HVs completed and entered the Initial MOD Fidelity Checklist 
and at least one follow-up checklist. They completed the initial checklist to document what PC 
TALK materials they provided to parents, how they described the concern about the child’s lan- 
guage, and how they reviewed the PC TALK strategies and encouraged parents to use them dur- 
ing daily routines (e.g., meals, play, or book reading; see Buzhardt et al., 2010, for full list of 
checklist items). On the follow-up checklist, HVs documented the degree to which parents 
reported using the strategies and/or how much the HV observed the parents using them, any chal- 
lenges parents reported using the strategies, new routines in which the strategies could be used, 
and new strategies to begin using if necessary. At Stage 3 implementation, HVs completed all 
Stage 2 steps, plus they entered at least three follow-up ECI progress-monitoring assessments 
and viewed the MOD’s description of the child’s progress and recommendations (MOD Step 5). 
There was natural variation in the number of families whose HV reached each implementation 
stage: Stage 1, n = 29; Stage 2, n = 15; Stage 3, = 12. 
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Language Activity Report. The Language Activity Report is a measure of how HVs worked with 
parents on their use of the PC TALK strategies across both experimental groups. At least 2 months 
after their child scored below benchmark on the ECI, an assessor, blind to the family’s experimen- 
tal condition, asked the parent 11 questions regarding how their HV provided support for their use 
of the PC TALK strategies. These 11 questions covered three areas of support: (a) provided PC 
TALK materials: four questions about what PC TALK materials were provided to parents (e.g., 
“Has your HV given you a manual related to your child’s communication or language?”’); (b) 
described child language concern: two questions that asked how the HV described the concern 
about the child’s low ECI performance (e.g., “Has your HV talked with you about specific prob- 
lems or challenges you may have with your child’s language?”’); (c) supported parent use of PC 
TALK: five questions asking how HVs worked with parents to promote their use of the PC TALK 
strategies across daily routines (e.g., “Has your HV showed by example how to use some of these 
communication strategies with your child?”). Parents answered Yes, No, or Don t know to each of 
the 11 questions. For items that asked what materials parents received, the assessor showed par- 
ents examples of the materials to help with their recall. No other prompts were provided. 


Working Alliance Inventory—Home Visiting Short Form (WAI-S). Because there is evidence that the 
effects of early intervention can be moderated by the quality of the parent—educator relationship 
(Burchinal et al., 2000), we also measured HV and parent perceptions of their working relation- 
ship using the WAI-S (Horvath, 1994; Tracey & Kokotovic, 1989). The full WAI contains 36 
items in which therapists and their clients rate their working relationship on a 7-point scale across 
three subscales: Agreement on Goals, Collaboration on Tasks, and Therapeutic Bond. The items 
for the WAI-S were chosen from the full WAI by conducting a confirmatory factor analysis and 
selecting the four items from each of the three subscales with the highest factor loading (Tracey 
& Kokotovic, 1989). The Therapeutic Bond subscale has since been adapted for the home visit- 
ing context and includes these three items: “I believe the parent/home visitor likes me,” “I appre- 
ciate the parent/home visitor as a person,” and “The parent/home visitor and I have built a mutual 
trust.” Parents and HVs rated these items on a 7-point Likert-type scale ranging from | (rarely) 
to 7 (always). Hatcher and Gillaspy (2006) reported reliability estimates for the WAI-S ranging 
from .88 to .92 and statistically significant correlations between the Bond subscale and the Cali- 
fornia Psychotherapy Alliance subscales ranging from .73 to .53. 


MOD satisfaction. HVs who used the MOD completed an anonymous online survey of their sat- 
isfaction with it. On a scale of 1 to 5 (5 being the best), they rated the usability and helpfulness of 
MOD features described in Table 5. Usability referred to the difficulty of completing the steps, 
and helpfulness referred to the degree to which the steps helped the HV “make decisions about 
how to address a child’s language needs.” HVs completed satisfaction surveys during annual 
trainings. 


Analytic Approach 


Before analyzing data, we removed outliers defined as 3 SDs above the ECI mean trajectory for 
each month of age. As a result, five ECI observations (0.8%) were removed and did not reduce 
the number of children in the study. Missing data on measures of the parent-HV bond (9.6%), 
HV’s education (20.5%), and parents’ education (2.7%) were handled using multiple imputation 
by chained equations (White, Royston, & Wood, 2011). 

We used multilevel modeling (Hox, 2010; Raudenbush & Bryk, 2002) to address the lack of 
independence in our data, that is, ECl-repeated observations were clustered within children, 
and children were clustered with HVs. Three levels were initially considered: Level | units 
represented the repeated measures, Level 2 units represented children, and Level 3 units 


256 Journal of Early Intervention 40(3) 
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Figure 2. Growth on the ECI before and after eligibility for each group of children. 
Note. EC] = Early Communication Indicator. 


represented HVs. We then used interclass correlation (ICC) and cluster effect (CE; aka design 
effect) to determine the suitable levels needed (Peugh, 2010). ICC at child or HV level indi- 
cates the proportion of ECI variation that occurred across the units at that level, and a non- 
trivial ICC suggests the necessity of including that level in the analysis. CE is a function of 
ICC and average cluster size. According to Peugh (2010), a CE smaller than 2 can be ignored. 
Thus, we excluded the HV level due to its small CE (ICC = 0.01, CE = 1.03) and fit growth 
curve models of ECI with two levels: repeated measures (Level 1) and children (Level 2). The 
ICC of the two-level unconditional model was 0.32, and CE was 2.1. 

Children’s growth on the ECI was assumed to follow a two-piece trajectory explained by two 
linear slopes: one slope before and one after scoring below benchmark on the ECI. Also, because 
children began treatment at various ages depending on when they scored below benchmark, we 
used a common month “0” (intercept) which represented the month of age that each child scored 
below the ECI benchmark and began treatment (see Figure 2). Thus, in the analysis, Level 1 exam- 
ined the within-individual change across time, and Level 2 examined the between-individual dif- 
ference, capturing the mean and the variance, respectively, in the individual intercepts and slopes. 

To address the first research question (Model 1), we examined the effect of experimental sta- 
tus coded as | = EBP + MOD, versus 0 = EBP on ECI growth to predict individual intercepts 
and individual slopes before and after eligibility in Level 2. To rule out the influence of variation 
in child’s age at eligibility, child age was included as a covariate in the model. 

For the second research question, we evaluated the effect of variation in MOD implementa- 
tion on parent-reported HV support and children’s ECI growth, controlling for age at eligibility 
(Model 2) for the ECI. Because MOD implementation was defined by four levels (no MOD and 
the three MOD stages), we used three dummy variables to represent Stage 1, Stage 2, and Stage 
3 variation in MOD implementation, whereas the EBP-Only group was treated as the baseline 
comparison group. Our primary interest was the differences in the after-eligibility slopes. For the 
third research question, the following covariates were added to Model 2: parents’ education, 
HVs’ education, primary home language, and parent—HV relationship. This procedure created a 
series-nested model wherein the deviance difference test was used to evaluate the best-fitting 
model (Singer & Willett, 2003). The chi-square test was used to examine differences in parent- 
reported HV support based on the Language Activity Report (LAR). HV satisfaction with the 
MOD, Research Question 4, was addressed by analysis of means and SDs of HVs’ ratings of the 
MOD’s usability and helpfulness on the Satisfaction Survey. 
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Table 2. ECl-Weighted Total Communication Rate per Minute by Child Age and Group. 


EBP-Only MOD Stage | MOD Stage 2 MOD Stage 3 
Age M SD M SD M SD M SD 
6 1.8 0.3 3.7 4.0 1.9 
7 47 2.3 3.0 
8 2.9 1.5 1.7 0.2 5.0 43 
9 1.1 1.1 4.8 0.9 1.2 3.6 1.8 
10 3.5 1.9 1.7 0.7 4.2 0.5 
1 2.9 2.2 4.4 0.3 3.4 1.6 
12 5.0 1.9 5.1 2.2 3.6 1.1 4.7 1.2 
13 6.3 6.1 78 4.8 
14 4.3 2.1 3.2 1.5 3.3 1.0 
15 6.0 5.3 5.7 2.2 4.6 3.6 4.2 1.7 
16 4.9 2.5 8.8 4.5 4.2 
17 5.8 3.4 3.6 2.5 5.1 1.7 7.3 1.1 
18 4.9 2.2 48 2.0 5.5 3.7 6.1 3.1 
19 4.4 1.7 6.7 2.8 4.8 2.1 6.1 4.7 
20 6.8 3.9 5.5 78 2.4 9.7 2.6 
21 79 4.5 8.1 1.9 5.6 3.5 14.2 0.6 
22 7A 4.3 5.5 3.0 9.7 2.3 7.1 4.1 
23 8.4 5.8 8.1 1.5 5.5 2.7 9.3 4.4 
24 9.7 3.8 9.5 3.9 11.3 6.4 9.6 3.3 
25 7.6 4.7 5.4 3.5 79 3.0 10.7 1.6 
26 10.3 10.5 15.9 7.6 79 79 21.1 3.9 
27 10.6 48 4.2 1.9 2.3 2.5 78 
28 12.5 10.1 12.2 7.1 12.1 43 18.3 8.5 
29 12.4 8.4 10.5 4.1 14.8 2.7 14.1 5.6 
30 11.4 5.0 15.8 8.0 8.2 1.8 22.3 2.7 
31 15.8 11.2 12.2 5.6 8.6 6.7 25.1 5.5 
32 14.9 8.7 18.0 8.2 10.8 8.4 18.2 14.0 
33 9.8 5.9 19.7 12.2 13.2 4.0 26.5 10.9 
34 16.5 12.6 14.8 4.6 9.5 9.7 16.2 6.1 
35 17.5 9.5 12.3 3.5 13.5 75 27.8 3.3 
36 13.7 5.3 15.3 10.7 20.2 17.7 28.9 1.8 
37 14.2 8.2 15.3 7.2 14.3 27.3 6.1 
38 14.5 77 35.4 
39 12.3 4.7 11.8 14.3 
40 30.5 7.2 55.5 
4| 20.2 
42 14.3 
All 9.660 8.927 9.663 12.530 


Note. ECl = Early Communication Indicator; MOD = Making Online Decisions. 


Results 


EBP + MOD HVs conducted 315 ECI assessments for 66 children, and EBP-Only HVs admin- 
istered 324 ECIs for 80 children. Children with EBP + MOD HVs received a mean of 4.77 ECIs, 
and EBP-Only children received a mean of 4.05 assessments over the study. Table 2 provides 
means and SDs of children’s ECI scores for each month of age across all groups. 
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Table 3. Two-Piece Growth Curve (Model 1). 


Y SE p 
Level | model 
Intercept 5.76 0.40 .00 
ASebefore -0.04 0.12 74 
Ageafer 0.79 0.11 .00 
Level 2 model 
MOD 0.04 0.57 5 
A8ebefore X MOD 0.08 0.14 57 
Age.frer X MOD 0.23 0.16 16 
Age at eligibility 0.37 0.04 .00 
ASebefore * Age at Eligibility 0.02 0.01 .05 
Ag@afrer X Age at Eligibility 0.04 0.01 .00 
Deviance (df) 3,829.8 (16) 


MOD = Making Online Decisions. 


Research Question |—MOD Effect on Expressive Communication 


The first step was to fit Model 1 to children’s ECI scores as reflected in Table 3. Level 1 results pro- 
vided the overall fit of the model (i.e., across both experimental groups). The overall ECI mean 
intercept was 5.76 communications per minute per month (SE = 0.40, p < .01) for all children. The 
mean slope before eligibility was —0.04 responses per minute (SE = 0.12, p = .74), representing a 
flat growth trajectory on the ECI before children fell below benchmark. After eligibility, the ECI 
mean slope increased to 0.79 responses per minute (SE = 0.11, p < .01). The Level 2 analysis intro- 
duced the effect of experimental status, the difference between the EBP + MOD versus EBP-Only 
groups. There were no differences in intercept (y = 0.04, SE = 0.57, p = .95) or slope (y = 0.08, 
SE = 0.14, p = .57) before treatment, indicating equivalence between groups at the onset of treat- 
ment. However, both groups improved their rate of growth after treatment. The EBP + MOD group 
demonstrated stronger growth (y = 1.02 [0.23 + 0.79]) than did the EBP-Only group at y = 0.79, 
but this difference (y = 0.23) was not statistically significant (SE = 0.16, p = .16). 


Research Question 2—Effect of MOD Implementation Stages on Expressive 
Communication 


Parent-reported HV support varied by LAR subscale. On the Provided PC TALK Materials sub- 
scale, 55% and 50% of EBP-Only and EBP + MOD parents responded Yes, respectively. On the 
Described Child Language Concern subscale, 34% and 30% of EBP-Only and EBP + MOD 
parents responded Yes, respectively. On the Supported Parent Use of PC TALK subscale, 28% 
and 23% of EBP-Only and EBP + MOD parents responded Yes, respectively. A chi-square test 
found no significant between-group differences on any of the three subscales: Provided PC TALK 
Materials, x?(2) = 3.09, p = .21, Described Child Language Concern, x?(2) = 2.67, p = .26, 
and Supported Parent Use of PC TALK, x?(2) = 4.96, p = .08. 

To examine the impact of variation in MOD implementation on children’s expressive com- 
munication (ECI scores), we first introduced implementation stage into Model 2 (Table 4). Again, 
controlling for age at eligibility, the Level 1 mean intercept was 5.78 communications per minute 
(SE = 0.39, p < .01) for the EBP-Only comparison group. Before age of eligibility, their com- 
munication rate slope was —0.02 communications per minute per month (SE = 0.12, p = .88). 
After the age of eligibility, the slope increased to 0.79 communications per minute per month 
(SE = 0.11, p < .01). Figure 2 shows ECI growth trajectories across groups. 
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Table 4. Two-Piece Growth Curve (Model 2). 


Y SE p dat 6 months dat 12 months 
Level | model 
Intercept* 5.78 0.39 .00 
ASerefore -0.02 0.12 88 
Ageater® 0.79 0.11 .00 
Level 2 model 
Stage | 0.50 0.76 Jl 
Stage 2 —1.32 0.81 .10 
Stage 3 1.39 0.91 13 
Ag pefore X Stage | -0.03 0.28 Il 
Ag@refore X Stage 2 -0.18 0.33 57 
Ag@pefore X Stage 3 0.17 0.15 .26 
Ageafer X Stage | 0.02 0.21 94 0.01 0.03 
Ageafer X Stage 2 0.16 0.21 43 0.13 0.25 
Agearer X Stage 3* 0.73 0.24 .00 0.56 1.12 
Age at eligibility* 0.37 0.04 .00 
Agebefore X Age at Eligibility 0.01 0.01 ai 
ASeafter X Age at Eligibility* 0.04 0.01 .00 
Deviance (df) 3,816.1 (22) 
> < 01. 
Table 5. Home Visitor Ratings of MOD Features. 
Usability Helpfulness 
MOD survey item M SD M SD 
Step |: MOD alert that the child has fallen below benchmark 4.63 0.59 4.44 0.81 
Step la: HV validation of the child’s low ECI score 4.54 0.65 4.33 0.78 
Step 2: Questions about why the child might be performing 4.50 0.63 4.35 0.78 
low on the ECI 
Step 3: Choosing the MOD-recommended PC TALK 4.28 0.79 4.46 0.75 
strategies and daily routines 
Step 3a: Printout of PC TALK strategies 4.09 1.10 4.34 0.92 
Step 4a: The Home Visitor Fidelity Checklists 4.18 0.91 3.98 0.94 
Step 4: The Parent/Caregiver Checklists for caregivers to 4.04 0.94 3.75 1.06 
show how often they use the strategies 
Step 5: The MOD report that shows the child’s performance 4.42 0.96 4.44 0.88 


before and after the intervention 


MOD = Making Online Decisions; HV = home visitor; ECl = Early Communication Indicator; PC TALK = Promoting 


Communication Tools for Accelerating Language in Kids. 


The Level 2 analysis estimated the effects of MOD implementation stage. Overall, MOD 
implementation produced significant differences in ECI growth. There were no differences in the 
growth trajectories before intervention. However, differences emerged after intervention, with 
the largest produced by the highest level of MOD implementation (Stage 3) compared with the 
EBP-Only group (y = 0 .73, SE = 0.24, p < .01). As shown in Table 4, Cohen’s d effect sizes 
for this group were 0.56 (medium to large) 6 months after treatment and increased to 1.12 (large) 


at 12 months. 
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Research Question 3—Influence of HV and Parent Characteristics 


To examine potential moderating influences of ancillary variables on these effects, we added four 
additional covariates to Model 2: HV education, parent education, primary home language, and 
parent-HV relationship. Parents’ mean rating of their relationship with their HV on the WAI-S 
bond subscales was 19.77 (SD = 2.53), with 21 being the highest possible rating, and HVs’ mean 
rating of their relationship with parents was 18.95 (SD = 2.53). Results of the deviance statistic 
difference test for nested models indicated that model fit was not significantly improved by add- 
ing parent education (p = .47), HV education (p = .27), primary language (p = .66), or bond 
rating (p = .63). 


Research Question 4—HVs’ Satisfaction With MOD Features 


Table 5 shows means and SDs of HV ratings for each of the eight satisfaction survey items. All 
features were rated at least a 4 (5 being the highest) on usability, and only two features were rated 
lower than 4 on helpfulness in guiding language intervention services: the HV Fidelity Checklists 
(3.98) and parent/caregiver checklists (3.75). In terms of usability, the MOD alert informing HVs 
that a child’s ECI score was below benchmark received the highest mean rating (4.63), whereas 
the optional parent/caregiver checklists had the lowest mean usability rating (4.04). For helpful- 
ness, choosing MOD-recommended PC TALK strategies was rated the highest (4.46), whereas 
parent/caregiver checklists were rated the lowest (3.75). 


Discussion 


We investigated the effects of implementation fidelity of an adaptive, data-based decision-mak- 
ing support tool on infant and toddlers’ growth in expressive communication. Because of the 
unique challenges associated with intervention delivery in home visiting contexts such as limited 
resources in homes, scheduling/missed appointments, and high staff turnover (Peacock et al., 
2013; Zercher & Spiker, 2004), implementation fidelity in this context is particularly relevant. 
We found that full implementation of the MOD led to the strongest growth in children’s expres- 
sive communication, when controlling for parent education, HV education, primary language, 
and parent—-HV relationship. These findings have substantial implications for early childhood 
services considering (a) the large-scale use of home visiting services; (b) the known links between 
early language development and later key outcomes such as school readiness (Hoff, 2013; Walker 
et al., 1994), academic success (Snow et al., 1998), and social competency (Doctoroff et al., 
2006; Miles & Stipek, 2006); and (c) growing demands for early childhood programs to use child 
outcome data to inform their services (Akers et al., 2015; Division for Early Childhood, 2014; 
Guss et al., 2013). 


Effect of Home Visiting Services and Progress Monitoring on Expressive 
Communication 


Children across both experimental groups demonstrated significantly stronger ECI growth after 
meeting the criteria for inclusion and receiving the evidence-based intervention. HVs in both 
groups were certified to administer and score the ECI quarterly for all children on their caseload, 
and all HVs were trained on the use of the PC TALK strategies with parents of children who 
scored below benchmark. The increase in children’s ECI growth from —0.02 before falling below 
benchmark to 0.79 after demonstrates the statistically and clinically significant impact of EHS 
HVs’ services on their children’s expressive communication when they monitor children’s prog- 
ress and provide intervention accordingly. 
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When we compared ECI growth between children whose HVs had access to the MOD’s addi- 
tional data-based decision-making guidance (EBP + MOD) and those who did not (EBP only), 
children served by EBP + MOD HVs showed stronger growth on the ECI, although this differ- 
ence was not statistically significant (Research Question 1). While this was somewhat unex- 
pected given prior findings (Buzhardt et al., 2011), some key differences between this study and 
the previous study likely contributed to these differing outcomes. First, the EBP-only HVs in the 
current study received annual onsite training on the use of the ECI and PC TALK strategies. In 
the prior study, HVs who did not use the MOD were given the PC TALK Manual and reviewed 
the strategies during an initial training without structured follow-up. Having this structured 
annual follow-up training provided HVs who did not use the MOD with additional support that 
the comparison group did not receive in the prior study. Second, because this was part of a larger 
study that involved additional child assessments, research staff contacted HVs to request consent, 
thereby alerting EBP-Only HVs when a child scored below benchmark. In the prior study, the 
only way EBP-Only HVs knew whether a child scored below benchmark was by viewing their 
child’s ECI graph. Therefore, EBP-Only HVs in the current study received some initial data- 
based decision-making support. 


Effect of MOD Implementation on Expressive Communication 


An important finding of this study was that children’s language growth was significantly stronger 
than EBP-Only when HVs fully implemented the MOD. These findings are similar to other stud- 
ies that have shown strong links between implementation fidelity and child outcomes (Dunst 
et al., 2013; Odom, 2009; Walker, Atwater, & Bigelow, 2011). However, like any data-based 
decision-making process, using the MOD adds to the complexity of services, requiring additional 
time and effort by service providers, or a cost at the expense of other services and activities with 
families, which is known to affect fidelity, sustainability, and acceptability by practitioners 
(Bywater, 2014; Carroll et al., 2007). 

MOD Stage | is considered a relatively /ow cost stage because the steps can be completed 
in about 10 min. MOD Stage 2 is a medium cost stage, requiring the HV to discuss the 
selected PC TALK strategies with the parent, provide guidance on their use of the strategies, 
complete the Initial MOD Fidelity Checklist and at least one Follow-up Checklist, and enter 
these checklists into the MOD system online. HVs who reached Stage 3, a high cost stage, 
completed at least three follow-up ECIs to monitor the child’s response to the parent’s use of 
PC TALK strategies, and the MOD provided a text-based description of the child’s progress 
immediately after entering the third and subsequent ECIs. Although Stage 3 is a high cost 
stage, children whose HVs achieved this level of implementation showed significantly stron- 
ger growth on the ECI compared with children whose HVs did not have access to the MOD 
or only implemented part of the MOD (Stage | or Stage 2). Despite the costs associated with 
each MOD stage, HVs rated all MOD features’ usability and helpfulness relatively high, with 
completion of Fidelity Checklists receiving the lowest ratings. Thus, the benefit of using the 
MOD may outweigh the time cost. 

Similar to findings from the prior MOD study (Buzhardt et al., 2011), the effect of MOD Stage 
3 implementation on child language growth grew over time as evidenced by the increase in effect 
size from a Cohen’s d of 0.56 at 6 months posttreatment to 1.12 at 12 months. This suggests that 
ongoing monitoring of children’s language by HVs and follow-up with parents about their use of 
the PC TALK strategies resulted in sustained use of the strategies and continued growth in 
expressive communication. Children’s growth on the ECI compounded over time during this 
12-month period relative to children whose HVs did not provide ongoing monitoring and follow- 
up with parents. 
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Influence of Other Variables on MOD Effect 


We found that regardless of family home language, parent and HV education level, and their 
reported relationship and appreciation for each other’s efforts, the effects of the MOD implemen- 
tation on ECI growth were maintained. This is a particularly meaningful finding because HVs in 
EHS and the parents they serve often have limited education backgrounds (Raikes et al., 2006), 
which may contribute to challenges in using data to inform intervention decisions and parents’ 
ability to follow through with recommendations. Without foundational knowledge in how to use 
data to inform decisions (e.g., interpret graphs, use trends to predict future performance, compare 
individual child data to normative trajectories), it is reasonable to expect that using data to inform 
the use of parent-delivered language strategies would be challenging without guidance or support 
either through ongoing technical assistance from a local expert or through a decision-making 
support system, such as the MOD. However, these findings must be considered with caution 
because we asked parents and HVs only to report their level of formal education, which may have 
missed in-service or technical professional development experiences specifically related to data- 
based decision making or program individualization. 

Although the quality of the HV—parent relationship is critical to the quality and fidelity of HV 
services in general (Barak et al., 2014; Brookes, Summers, Thornburg, Ispa, & Lane, 2006; 
Korfmacher et al., 2007), we found that the effectiveness of MOD Stage 3 implementation was 
not significantly related to parent- and HV-reported relationship ratings. This is promising 
because it suggests that the benefits of using the MOD may not be dependent on a strong HV— 
parent relationship per se. However, this finding should also be considered with caution given the 
limited variation in the parent—HV ratings. On average, parents and HVs rated each other’s rela- 
tionship as high. To increase our confidence in this finding, further research is needed with a 
sample that has more variance in HV-parent relationships and perhaps with additional relation- 
ship measures that are not based on self-report. 


Limitations 


Key limitations of this study are the use of the HV-administered ECI as a child outcome mea- 
sure and the use of parent report (i.e., the LAR) to measure how much support HVs in both 
groups provided parents on their use of the PC TALK strategies. Future research should assess 
child outcomes using a standardized measure administered by independent assessors to mea- 
sure the degree to which these findings generalize to similar measures. Regarding parent 
report, the PC TALK strategies are just one among many services and supports that HVs pro- 
vide families, and considering the LAR was given at least 2 months after the child initially 
scored below benchmark on the ECI, there is concern about the validity of parents’ LAR 
responses because they may have forgotten specific activities that the HV did with them. Valid 
and reliable measurement of implementation fidelity can be challenging in applied studies 
(Gearing et al., 2011; Lloyd, Supplee, & Mattera, 2013; Schoenwald et al., 2011; Supplee & 
Metz, 2015), but it is particularly challenging in home visiting studies that take place in geo- 
graphically remote homes with home visiting schedules that can sometimes be erratic (Paulsell, 
Del Grosso, & Supplee, 2014). In addition, using independent observers to code fidelity during 
home visits may cause a Hawthorne Effect, in which the presence of independent observers in 
home settings may influence the behavior of HVs and parents. In Paulsell, Del Grosso, and 
Supplee’s (2014) review of 178 studies on home visiting programs, only 22 studies reported 
implementation fidelity data, and only four of those studies used observational fidelity mea- 
sures. Advances are needed to support feasible methods of collecting valid and reliable data on 
HVs’ implementation of EBPs. 
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Future Directions 


Further work is needed to improve the uptake of data-based decision-making practices by infant— 
toddler programs. For example, future MOD advancements would allow programs to use the 
MOD to document implementation of their existing curricula, rather than tying the guidance to 
an intervention that they are not already using. Although some curricula and EBPs used by 
infant-toddler programs support some form of progress monitoring and decision making using 
curriculum-based measures, the MOD provides meaningful data about a child’s progress on a 
general outcome (i.e., communication) rather than outcomes on precise skills or competencies 
targeted by the curriculum (e.g., responds to name, turns the page of a book, points to show inter- 
est or direct attention) 

Data-based decision-making practices should be expanded beyond language outcomes. 
Currently, the MOD’s guidance targets only communication because its algorithms are designed 
to analyze ECI data to guide decision recommendations. In addition to the ECI, the Infant and 
Toddler IGDI online data system in which the MOD is housed accepts data from the Early Social 
Indicator, Early Problem Solving Indicator, and Early Movement Indicator (Buzhardt & Walker, 
2010). Similar to the ECI, each of these measures includes age-based benchmarks (Carta et al., 
2010), so the MOD could be expanded to provide real-time guidance informed by any one of 
these measures or a combination of multiple measures. 

Further research is also needed to develop supports and/or tools that further reduce the costs 
of implementing these practices, particularly those high cost and less preferred steps, and moni- 
toring their use in home visiting contexts. For example, additional implementation supports 
could include automated reminders/prompts (e.g., via email, texts, smartphone notifications) for 
educators to follow up with parents and administer ongoing assessments at scheduled times. 
Also, mobile devices could be used to complete and enter MOD checklist data during home vis- 
its, rather than using paper checklists that must be entered into the MOD after the home visit to 
increase efficiency. Finally, structured follow-up training or technical assistance that targets spe- 
cific implementation deficiencies, particularly during the early stages of implementation, may 
benefit programs that struggle to sustain full implementation. 

These findings suggest that infant-toddler home visiting programs would likely benefit from 
implementing a data-based decision-making model informed by progress-monitoring outcomes. 
However, the question remains for the early childhood field in general: How do we further pro- 
mote and sustain high fidelity data-based decision-making practices by infant—toddler programs 
already challenged by insufficient budgets, limited resources to support these practices, and staff 
with limited training and expertise in data-based decision-making practices? Technology clearly 
has a role in improving the feasibility of using data to inform services, but careful attention is 
needed to ensure that the benefits of these practices, whether using the MOD or another model, 
justify service providers’ time and effort needed to implement them. 
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